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Motion capture-driven simulations? 
Motivation: 



flivcr/idc Graphic/ Lab 



Motion capture is already the industry standard 
for lifelike, 3D characters 

Physical 'ragdolls' and engines are gaining in use 


Motion capture-driven simulations? 
Motivation: 
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As the cost of simulation computation goes down 
and demand goes up, we will see a tighter coupling 
of the simulation and motion capture techniques 


Examples of blending are already appearing (Havok2) 


What are mocap-driven simulations? 
Dynamically simulated characters 
that follow motion capture, actively 



flivcr/idc Graphic/ Lab 


Motion capture data 


Dynamic 

model 


Final animation 


Why use mocap-driven simulations? 

To get the best compromise between: 
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Human motion capture 

+rich with style & detail 
- hard to adapt or to be made 

to 'respond' to new scenarios 


Dynamic simulation 

+physically realistic 
+handles a changing environment & 
can 'react' in believable ways 
- requires a controller to actuate 




Respond to new scenarios? 

A changing environment? nivcr/idc qrapbic/ Lab 

Reacting in believable ways? Huh? 




Why do we want realistic reactions? 


Beyond ’ragdolls’ that 'play dead', want 
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characters that take a lickin ' and keep on tickin ' 




Overview: 
System Layout 
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mocap 

handler 


upper body 
controller 


graphics 


dynamic 

model 

Integrator 








Overview: 

Building a reactive character 
Tracking Balance Control 
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Tracking Control 


Converted 

desired 

Mocap Data 

joint angles 



flivcr/idc Graphic/ Lab 


T racking 
Control 


computed 

Dynamic 

torques 

Model 




Tracking Control 
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Equations of motion - computed by automatically (SD-Fast) 



Boxing sim 
no wrists 


(39 dof) 


X 


y 





Tracking Control 


Converted . 


Tracking 


Dynamic 

Mocap Data 


Control 


Model 
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PD-servo controller computes torques 



T=k(0,-e)-b(0) 


0^ from motion data 

k and b are uniform 
stiffness and damping 

Note: No joint limits, instead 
influenced by data 






Tracking Control 
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Inertia scaling for stiffness and damping 



k and b are scaled by 
moment of inertia: 


k = k’ * MOI effect 
b = b’ * MOI effect 


tune for uniform k and b 
Then: 

high stiffness + moderate 
damping = good tracking 






Tracking Control 


Converted . 


Tracking 


Dynamic 

Mocap Data 


Control 


Model 
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Convert raw motion capture data to joint angles 

Optical: map/fit to skeleton 


^ inboard 
marker 

outboard 
yT marker 


Electromagnetic: preprocess 
using marker orientation 
data for joint angles as 


® desired ® 


out 


Then for both, fit spline thru 
samples (sim 'prefers' such 
smoothed inputs) 







shoulder an^e XYZ (rad) 








T racking control is flexible enough to 
follow a large variety of motions... 
...from the waist up 
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How about the rest of the body? 
Need lower-body control 
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Tracking Balance Control 




Lower-body Control 
Balanced standing 
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Controller's goal: 

Keep the simulation's 
center of mass (com) 
safely inside the 
support polygon made 
by the feet 

To accomplish the goal: 

Pick a desired com 
and minimize errors by 
making corrections in 
the leg actuation 



Lower-body Control 
External balance force 
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First compute the required pelvis foree 
that would result in balance, but don't 
apply it directly... 


Balancing force to control 
center of mass: 






Lower-body Control 
Virtual actuator method 
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Inspired by 
Pratt (1995) 

Convert force to torques for 
virtual actuator: 


T _ Jt ®M 

^balance “ ‘''‘(/i -> a) 

^ track + '^balance 






Lower-body Control 

Using the motion capture data 
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Add in info about the 

action taking place 
by extracting data 
from the mocap: 

Desired as estimate com: 


E ni/ (^marker u 

mtotal 



Also, track the data in hips, knees, ankles 
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Full-body mocap-driven simulations 







Full-body mocap-driven simulations 



0.20 : 


com estimated 

0.00 : 




' 


- 0.20 

Human data 


simulation com 



Virtual actuator method 
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Comparison for dancing motion (sim in blue from previous slide) 
normalized from one foot to the other on the horizontal 





Full-body mocap-driven simulations 



Footwork is nice, but lets see some contact! 
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Overview: 

Control for hitting and reacting 
Tracking Balance Control 



flivcr/idc Graphic/ Lab 



Control for acting and reacting 



Continuous play 
state machines 


Control over actions 


Reacting to contact 
collision forces 
gain scheduler 
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Control for continuous play 
Interpolation finite state machines 
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Transitions interpolate (slerp) from one mocap clip to the next 



Control for (upper body) actions 
Editing motion capture, as usual 
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Use motion capture 
library of examples 
(swings, punches, etc.) 


Interpolation, IK, 
and warping, etc. 
for parametric 
control 


Control for actions 
Edit clips for position and orientation 
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Use IK to hit target 
Apply IK offsets: 


offset “ 0//f “ 0a(t//f) 


Offsets smoothed 
further by dynamics 





Control for actions 

Build new examples 'on the fly ' 
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Interpolate with any 

constant value y 
to get an in-between 
action 

Time-warp to align 
important features 
in time: like start, 
target pt (hit point 
furthest extent, etc), 
and end 



0 <7 < 1 




Control for actions 

Speed-up or slow-down only 
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Speed of end-effector 
relies on angular velocity: 


n joints 

m = 5^ r; X 0);(t) 
1 = 0 


Preprocess to find 
unmodified speed 

Thenm time-scale 
by oc-i at hit time 



scaled 


1 


region 


sinusoid Y(t) 


t' 


0 







Control for table tennis simulation 
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Control for boxing simulation 
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Control for reacting to contact 
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Control for reacting to contact 
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Dynamic impact 
adds external forces 
to the simulation 

Collision handler 
detects and computes 
penalty force reaction 

Apply reaction forces 




kinematic playback 

dynamic (no collision) 

dynamic (collision) 

raw position data 



drum 

surface 


0.7 

time (s) 



Control for reacting to contact? 
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React to forces 
Recover smoothly 


Gains k(t), b(t) 


Lower gain to avoid 
stiff contact, allows for 
bigger timestep (overall 
speed-up) 




X angle (rad) 


Control for reacting to contact 


flivcr/idc Graphic/ Lab 



Creates a nice 
smooth space 
(as shown) to 
give good 
handle for 
desired affect 


Stiff or loose-looking character can both result, based on tuning 



Evaluation: real vs. simulation 
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the end, right? 


No wait, there’s more: 

TRICKS and CHEATING 
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Okay, so sims are great, but... 

How do we make them easier to contol? 

Give up some (small amount) of the realism! 


How do we make them fast(er)? 

Give up some (more) of the realism! 

Do we really need to simulate a full body? Always? 

Only have to simulate what is to move based on 
dynamic effects, the rest can just come along for 
the ride (kinematically.) Likewise, only need to 
simulate when these affects are actually needed 


Speed-ups: 

Simulation speed relies on several factors 

But they boil down to two: 

Timestep & Compute-time/per cycle 

Factors that can affect these: 

Integration method -> implicit solvers can take 
bigger steps in general (but may look 
over-damped... the tradeoff!) 

Methods for solving constraints, especially for 

resolving contact -> avoid rigid constraints 
to avoid the need for tiny timesteps 

Number of body parts -> the fewer, the faster 
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Ultimate speed-up: Only simulate what 
you need, when you need it! 
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Turn off the sim (change to kinematics) and back as needed, 
can result in amazing speed-ups, but need to make good 
switches between representations 

Shapiro and Faloutsos (’03) offer some answers 

Use level-of-detail to simulate only needed motion and 
complexity (and cull when off camera) 

Carlson and Hodgins (’97) discuss this topic 

Simulate only the arm or leg (or whatever) in contact and 
use the kinematics and mocap for the rest {hybrid model) 
(Already seeing this in some games!) 


How do we make control easier? 

CHEAT (on the physics that is) 

Once the academics wash up and go home, 
developers are left to fill in the details 
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Physics in games only needs to be used when it 
adds to the look or gameplay. And nobody requires 
developers to 'play by the rules' so... 


How about for starters, lets avoid torques (So 
unintuitive!) & apply forces, any force will do (legal or not) 


And, why do real balance control (Hard!) when there 
are perfectly good fake balancers that are easier to 
control and can result in 'pretty real'-looking motion? 


Shameless plug: We've worked on using 
a sim to map data to new characters while 
adding in ground forces (Zordan & Horst 03) 


Optical data + Simulation Posture 
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Use this same technique for: 
Force-based control 
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The technique controls the sim to move 
'like' the actor based on the mocap, by 
attaching the mocap markers to the 
landmarks on the sim using springs and 
dampers 

This method makes controlling 
easy but doesn't guarantee good 
reactions... must manage separately 


Force-based control 
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Matching virtual 'landmarks' guide the simulated bodies 
to follow the markers using intuitive forces 



^marker 


Frf, 


amping 


Springs pull the simulation to 
the marker data 



Body forces damp motion 








CHEATING in lower-body control: 

Use an external balancing force 
("Hand of God" van de Panne 95) 
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If the force only gets applied horizontally 
the sim will be standing on its own but just 
won't be "balancing" on its own 

Cut the force when it gets too large and the 
sim will fall, ramp it down, cap it, plenty of 
options here to get 'the right look' 


= (err)-b,(efr) 





CHEATING in lower-body control 

Or glue one foot (or both) to the ground 

If one foot is hxed to the ground, the whole 
body will move but it won't fall. Gravity 
can still act & look right as long as the 
other foot can contact the ground 

Let the 'glued' foot pivot on the ground 
for further freedom, or add a spring to 
mimic ankle activation 

Again turn the glue off when things are 
'out of balance' and let the sim fall over 





Incidentally, this kind of CHEATING 
doesn't mean it won’t be realistic... 


Biomechanists study balance/falls this exact way: 



(Hsai, 99) 


with a spring between the ankle and the ground! 
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Can use simple active control to 'catch' or prevent falling 
Also could use the upper body for balance, too 
waving arms, etc. 



Conclusions 
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Motion capture and dynamics are a powerful 
combination but does not solve the whole 
control problem 

Hybrid dynamics/kinematics approaches 
will likely beat out pure dynamics alone 
because they provide robust control and 
'unreal' results 


